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The present invention provides apparatus and methods 
- for a multi-dimensional user interface for use in audio 
visual production. A display system including a central 
processing unit (CPU) is coupled through appropriate 
input/output (I/O) circuitry to input dcvdces, such as a 
keyboard, digital pad and/or track ball as well as a 
display device. The CPU is further coupled to a hard 
disk drive for the storage of programs and data, and is 
also coupled to a network through which the CPU may 
conmiunicate with a variety of system resource devices 
such as editors, music synthesizers, graphics generators, 
scheduling resources, audio enhancement resources, 
etc. A user viewing the interface on the display may 
utilize one of the input devices, such as by way of exam- 
ple, the keyboard, to select, incorporate or otherwise 
integrate the various system resources to develop a 
unified multi-media production. The user interface of 
the present invention includes a control frame which in 
practice substantially fills all of the display screen of the 
display and is consistent for all user applications. The 
control frame is comprised of control panels which 
surround a variety of subwindows and acts as a consis- 
tent control area for all users of the interface. Once 
defmed, elements may be selectively placed on an event 
horizon bar in the control frame. The placement of an 
element on the event horizon results in the display of 
timing data for the clement, relative to other elements 
on the event horizon. 



56 Claims, 7 Drawing Sheets 
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composers, and the like may provide both real time and 

MULTI-DIMENSIONAL USER INTERFACE non-real time input to exchange necessary project data, 

and effectively coordinate the production of Uie entire 

BACKGROUND OF THE INVENTION ^ media work. Accordingly, one of the requirements of 

1. Field of the Invention ^ ^^V common multi-media user interface is the ability to 
The present invention relates to apparatus and meth- integrate multi-media types, and to provide the operator 

ods for displaying, manipulating, and selecting multi- with the ability to manage large quantities of informa- 

media or computer stored information in a window tio" in an understandable and efficient manner. The user 

environment of a computer display system. More par- interface must be intuitive and flexible to acconunodate 

ticularly, the present invention relates to an improved a variety of operator editing styles and personalities, 

user interface to provide a unified operator interface for For example, a music composer who thinks typically in 

a wide range of systems which must be coordinated and terms of scores^ notes and related music timing, should 

monitored in a multi-media production system. be able to work in that environment using a standard 

2. Art Background user interface, and not be required to work in terms of 
Videotape editing environments have evolved from video time code or other non-music related external 

providing simple editing cuts to the incorporation of standards. Similarly^ the film production director or 

full featured graphics, film to tape and other processes special effects engineer should be able to utilize the user 

to complete a video production. Consequently, com- interface in a manner consistent with their work envi- 

puter controlled editing systems and integration meth- ronment, which may, by way of example, be illustrated 

ods have been used to incorporate and integrate various through the use of video time code signals (and not 

production media resources such as special effects, mu- music). 

sic graphics and the Uke. However, due to the nature of as will be described, the present invention provides a 

film and video production, a variety of resources must multi-media user interface which may be utUized by a 

be integrated, scheduled and coordinated with one an- variety of multi-media professionals in the production 

other to obtam a completed product. 25 ^.j^ ^^^^ Unlike prior art window based 

Histoncally. humaris have mterfaccd with computers , g ^^^3^ invention provides a three- 
through a system of discrete commands which typically dimensional representation (known as an "element") of 
comprise a comhmation of both text and mathematical infonnation to be manipulated. Each clement comprises 
symbolic characters Examples of such systems are ^ dimensional representation of a ••resource," A 
numerous and include Ae progj^ming languages of 30 ^ ^ ^ three-dimensional object which may have 
Fortran, Algol. Basic etc.. which transform a g^^^^ represented in either two or three-dimensional 
of user commands into machine executable object - * • j 4. » t.- 1. u 

r^TT -.1. t_- 1. i_ form. A wmdow compnses a venue which may be 

code. However, the case with which a user becomes ^ wutu^ w^i* f « » ^ 

proficient in programming or interacting with a com- ^r^ngx^r^d for specific activities, such as music produc 

puter-based system is generally a function of how close 35 ^^tu special effects scheduling and Uie hke. However 

the system models the logical thought of the user him- ^ "^tcrfacc shares common fundamental tools and 

self If the user is able to enter commands in the order in ^he same data base, such that each media professional, 

which he would find most logically appropriate, rather ^^'^ ^s. » videotape editor, audio editor, producer, etc. 

than having to transpose his desired command into the *e mterface m a consistent manner, 

code of a programming language, greater user effi- 40 SUMMARY OF THE INVENTION 

ciency in using the system is achieved. . .. 

A number of systems which have been developed to The vxestnX mvention provides apparatus and meth- 

minimize the learning or acclamation period which a * multi-dimensional user interface for use in 

user must go through to become proficient in the inter- a^idio visual production. A display system including at 

action with the computer system are referred to as "Ob- 45 least one central processing unit (CPU) is coupled 

jcct oriented systems". A common object oriented in- through appropriate input/output (I/O) circuitry to 

terface approach utilizes multiple "windows" displayed "JP^t devices, such as a keyboard, digital pad and/or 

on a cathode ray tube (CRT) in which combinations of ball. The CPU is further coupled to a hard disk 

text and graphics are used to convey information. Each drive for the storage of programs and data, and is also 

window may take the form of an object such as a file 50 coupled to a network through which the CPU may 

folder, different operating environment, pages or lay- communicate with a variety of system resource devices 

ered bit maps to provide a separate display of video such as editors, music synthesizers, graphics generators, 

information in independent screen regions. (Sec, for scheduling resources, audio enhancement resources, 

example, Robson. "Object Oriented Software Sys- etc. The CPU is also coupled to a display device (for 

tcms". Byte. August, 1981; and U.S. Pat. Nos. 4,414,628; 55 example, a CRT) on which the present invention's user 

4,533.910; 4,450,442; 4.555,775 and 4.622,545, and L. interface is displayed to the user. A user viewing the 

Tesler. 'The Small Talk Environment", Byte. August, interface on the display may utilize one of the input 

1981, Volume 6, No. 8.) devices, such as by way of example, the keyboard, to 

The use of modem computer systems incorporating select, incorporate or otherwise integrate the various 

object oriented window environments may be applied 60 system resources to develop a unified multi-media pro- 

to multi-media production methods, such as videotape duction. The user interface of the present invention 

editing, audio mixing, etc. However, one unique prob- includes a control frame which in practice substantially 

lem associated with multi-media production is the ne- fills all of the display screen of the display. The control 

cessity to provide the ability for a diversity of media frame is comprised of control panels which surround a 

professionals to collaborate and exchange project data 65 variety of subwindows. and acts as a consistent control 

in a consistent interface environment. By providing a area for all users of the interface. The control frame 

consistent user interface, media professionals such as includes a construction area which corresponds typi- 

special effects engineers, animation specialists, music cally to a front view port looking toward a three-dimen- 
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sional element which is a representation of a resource. The manipulations performed are often referred to in 

The control frame further includes a top view port terms, such as adding or comparing^ displaying, etc. 

which illustrates the time relationship between the vari- which arc commonly associated with mental operations 

ous resources in a "venue". Effectively, the control performed by human operator. No such capability of a 

frame of the present invention provides a two-dimen- 5 human operator is necessary, or desirable in most cases, 

sional window to selectively view a three-dimensional in any of the operations described herein which perform 

"element". In operation, a user specifies mandatory and part of the present invention. In the present case, the 

optional attributes which an element must have, and operations are machine operations. Useful machines for 

defines the element representing the resource within the performing the operations of the present invention in- 

construction area of the control frame. Once defined, 10 elude general purpose digital computers or other similar 

the element may be selectively "dragged" down to an devices. In all cases, there should be borne in mind the 

event horizon bar at which time, time data is displayed distinction between the method operations of operating 

in the time view port of the control frame. Using the a computer and the method of computation itself. The 

interface of the present invention, elements may be present invention relates to method steps for operating 

created, edited, bundled, integrated and rearranged 15 a computer graphics system and processing electrical or 

along the event horizon. other physical signals to generate other desired physical 

BRIEF DESCRIPTION OF THE DRAWINGS j^.^ntion dso relates to apparatus for 

FIG, 1 is a conceptual illustration of the present in- peiforming these operations. This apparatus may be 
vention's integrated multi-media user interface for inte- 20 specially constructed for the required purposes, or it 
grating and manipulating a variety of multi-media func- may comprise a general purpose computer selectively 
tions. activated or reconfigured by a computer program 

FIG. 2 is a functional block diagram showing one stored in the computer. The algorithms and displays 
possible data processing system incorporating the presented herein are not inherently related to any par- 
teachings of the present invention. 25 ticular computer or other apparatus. Various general 

FIG. 3 is a conceptual illustration of the present in- purT>ose machines may be used with programs in accor- 
vention's use of venues to represent data sets of re- dance with the teachings herein, or it may prove conve- 
sources available to a user. nient to construct more specialized apparatus to per- 

FIG. 4 conceptually illustrates the use of venues and form the required method steps. The required structure 
view ports by the interface of the present invention. 30 for a variety of these machines will appear from the 

FIG. 5 is a conceptual illustration of an element rep- deiscription bdow. In addition, the present invention is 
resenting a resource, as viewed in three dimensions not described with reference to any particular program- 
utilizing the present invention's user interface. ming language. It will be appreciated that a variety of 

FIG. 6 is a front view of a user display screen utiliz- programming langiiages may be used to implement the 
ing the present invention's multi-media user interface. 35 teachings of the invention as described herein. 

FIG. 7 is the same view as FIG. 6 except with the i^ry-r a tt ct^ e«» 

chnent attributes window opened for venue selection. DETAILED DESCRIPTION OF THE 

FIG. 8 is the same view as FIG. 7 further illustrating D^fVENTION 
the present invention's user interface in the display in a The following detailed description will be divided 
plurality of elements and venues selected for the pro- 40 into several sections. The first of these will treat a gen- 
duction of a multi-media work. eral system arrangement for generating computer 

FIG. 9 is the same view as FIG. 8 further illustrating graphics in accordance with the teachings of the present 
the element attribute block opened and venue and re- invention. Subsequent sections will deal with such as- 
Rource selection. pects of the present invention as the general conceptual 

NOTATION AND NOMENCLATUllP defmition of a 'Venue" and "resource", and the struc- 

INU 1 A 1 ION AND NOMENCLATURE ^ operation 6f the present invention's multi-media 

The detailed descriptions which follow are presented user interface, 
largely in terms of graphics interfaces, algorithms, and In addition, in the following description, numerous 
in symbolic representations of operations of data bits specific details are set forth such as functional blocks 
within a computer display system. These algorithmic 50 representing data processing devices, window configu- 
descriptions and representations are the means used by rations, etc. in order to provide a thorough understand- 
thosc skilled in the data processing arts to most effec- ing of the present invention. However, it will be obvi- 
tivcly convey the substance of their work to others ous to one skilled in the art that the present invention 
skilled in the art. may be practiced without these specific details. In other 

An algorithm is here, and generally, conceived to be 55 instances, well known circuits and structures are not 
a self consistent sequence of steps leading to a desired described in detail in order not to obscure the present 
result. These steps are those requiring physical manipu- invention unnecessarily. 

lationsofphysical quantities. Usually, though not neces- , 

sarily, these quantities take the form of electrical, or GENERAL. SYST1EM CONFIGURATION 

magnetic signals capable of being stored, transferred, 60 Referring now to FIG. 1, the present invention is 
combined, compared, and otherwise manipulated. It conceptually illustrated. As previously discussed, the 
proves convenient at times, principally for reasons of present invention provides an integrated user interface, 
common usage, to refer to these signals as bits, values, such that media professionals may utilize a conmion 
elements, symbols, icons, characters, terms, numbers, interface and integrate various production processes 
windows or the like. It should be borne in mind, how- 65 such as animation, special aflects. editing, mixing and 
ever, that all of these and-similar terms are to be associ- production scheduling. Through the use of the present 
ated with the appropriate physical quantities and are invention's user interface, a variety of resources may be 
merely convenient labels applied to these quantities. accessed in the production of a feature film, videotape 
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or the like. Unlike prior art systems in which separate ppcniTPrF ArrF<;<; XHTinTinH vfkiif^; 

special effects production facUities, live action facUities RESOURCE ACCESS THROUGH VENUES 

and music score editing facilities are required to work Referring now to FIG. 3, the present invention con- 
independently, and then be integrated at some later ceptually permits a variety of media resources such as 
date, the present invention provides a new editing con- S special effects 90» editing sequencer 92 and video re- 
cept to permit an operator to interact with each of the sources 95 to be viewed by a user 100 through a "vie^y 
resources comprising a multi-media production and port*\ relative in time, along an **event horizon" 104. As 
generate a final completed work. As v/ili be described will be described, a view port provides a perspective 
more fully below, the present invention accesses, ar- view of data contained in a "venue", wherein a venue 
ranges and coordinates these various production re- 10 may include a plurality of resources such as audio re- 
sources through the use of an integrated user interiace. sources 102, video resources 95, etc. 

Referring now to FIG. 2, one possible computer Referring novv to FIG. 4, the present invention's 
graphic system employing the teachings of the present concept of venues is disclosed in relation to three-di- 
invention is shown. As illustrated, the graphics system mensional space. The venue concept of the present 
includes a computer 20, which comprises six major IS invention allows the tiser 100 to view data represented 
components. The first of these is a central processing in alternative view pons of a venue. Relationships be- 
unit (CPU) 22 which is coupled to a memory 24. CPU twccn a variety of resources are apparently based on the 
22 and memory 24 are further coupled to a user inter- view which the user 100 chooses. As illustrated in FIG. 
face circuit 26 and a disk input/output (I/O) circuit 28. 4, a venue is a three-dimensional space in which object 
for communicating with a hard disk drive 30 for mass 20 "elements** reside. An "element" is a three dimensional 
storage of data. Computer 20 further includes a serial representation of a resource coupled to, for example, 
input/output (I/O) circuit 32 for communicating with the network 44 depicted in FIG. 2. Depending on the 
serial devices over line 34, such as by way of example a view port which is chosen, the elements identify them- 
digitizing pad 36, a keyboard 38, and a track ball input selves in various ways. For example, in FIG. 4, element 
device 40. Computer system 20 further includes a net- 25 110 may comprise a graphics resource, element 114 may 
work interface circuit 42, which is coupled to a network represent an audio resource, element 116 may represent 
44 and a gateway circuit 48, which permits the com- a textual planning resource, and element 118 may repre- 
puter system 20 to communicate with other comrouni- sent a music or other audio resource. Viewing a venue 
cation systems over telephone lines, optical fibers and through a top view port reveals the relative placement 
the like. 30 of the various elements in time, both the duration of 

In addition, a display monitor 50 is illustrated which time that an element represents (such as a film "take") 
is used to display the integrated user interface compris- and the relationship between the resources to a refer- 
ing the present invention, and is coupled to the user ence time, which may be a recording machine, fmished 
interface circuit 26. As also illustrated, a variety of film or other multi-media piece. Viewing the right side 
resource devices such as a video tape recorder (VTR) 35 view port reveals versions of the elements, for example, 
52, music synthesizer 55, audio tape recorder 60 and audio element 118 includes two versions; namely, ver- 
production switcher 62 are coupled to the network sion 120 and version 122, along with the relative time 
interface circuit 42 over the network 44, through device lengths of the versions. In addition, it should be noted 
translators 64, 66, 68 and 70, respectively. that the user interface of the present invention permits 

As will be described more fully below, the present 40 the use of multiple venues, wherein some or all of the 
invention as illustrated in FIG. 2 permits data from venues may share the same time period, and each venue 
resources such as VTR 52, music synthesizer 55 and includes its own elements. 

audio tape recorder 60 to be coupled to the present For sake of example, assume that the elements illus- 
invention's user interface. A user viewing the interface trated in FIG. 4 represent resources for the production 
on display 50 may utilize one of a variety of input de- 43 of a soap opera. In this example, assume that a produc- 
viccs, such as by way of example, keyboard 38 or track tion crew photographed a variety of scenes, transferred 
ball 40 to select, incorporate and otherwise integrate the these scenes to video tape, and that these scenes corn- 
various system resources to develop a unified multi- prise scene 1, take 1, etc. 

media production. As shown in FIG. 5, viewing an element through the 

It will be appreciated that the present invention, as 50 front view port reveals the type of resource through the 
illustrated with reference to FIG. 2, is only one possible use of an icon label. For example, in the case of an audio 
embodiment of many. For example, although only one resource, the icon label may comprise a graphic repre- 
computer 20 is shown, the present invention may in- sentation of a musical note. In addition, the various 
elude multiple CPU's and/or computers coupled to the versions are iUustrated, as in the case of element 118 in 
network 44i Each of these CPU's and/or computers 55 FIG. 4, by darkened lines traversing the longitudinal 
coupled to the network 44 may execute separate pro- length of the rectangular element image with respect to 
grams, and support different versions of the user inter- FIG. 5, the version may be activated, and thereby run, 
face of the present invention. Alternatively, it is con- by placing a cursor on the screen of display 50 over one 
templated that the use of multiple computers and/or of the activation buttons 130 on the element, and pro- 
CPU's may support a system of distributed computing, 60 viding an activation signal, such as for example from a 
wherein multiple versions of the present invention's "mouse", to run that particular version of the resource, 
user interface may be supported concurrently, each of Returning to the present example of a soap opera pro- 
the multiple user interfaces executing different pro- duction, versions 120 and 122 of the audio element 118 
cesses but having access to a common data base of re- as shown in FIG. 4 and versions 135 of the element 140 
sources. As such, it will be noted that the embodiment 65 as shown in FIG. 5 may comprise musical options for 
shown in FIG. 2 is a simplified embodiment for pur- use during the particular time associated by the length 
poses of illustration, and is not meant to limit the present of the elements. Similarly, if the resource comprises 
invention's utility. scenes, then, as is common in the industry, each of these 
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scenes may have associated time codes (such as port 215 is also provided in control frame 150. As previ- 
SMPTE) which comprise the time in and out for each ously illustrated in FIG. 4, elements representing re- 
scene. Accordingly, by viewing the element represent- sources may be viewed and operated upon by appropri- 
ing the resource in terms of time, the time length of each ately selecting a view point. Although the control frame 
scene would be represented on the element by the 5 150 displays objects in two dimensions, by appropri- 
length of the particular version lines, for example lines ately selecting the view port, the elements may be 
135. or alternatively, by duration lines 140 defining the viewed from all three dimensions. Elements, utilized in 
beginning and end time of each scene of the resource. the present invention, directly represent the objects that 
INTEGRATED USER INTERFACE "P ^ production, such as scripts, segments of 

10 video tape, score, scenes, director notes, sound tracks, 

Refernng now to FIG. 6, the integrated user inter- etc.. and arc identified by an icon on the front face of 

face of the present mvcntion wUl be described. As the element as previously described with respect to 

shown in FIG. 6. the present invention's user interface piQ. 5. Elements can be viewed from the from. side, or 

mcludcs a basic conuol frame 150 which m practice fills the top, or in multiple views. As previously dis- 

substantially aU of the display screen of display 50 as 15 cussed with reference to the concept of "resources", the 

depicted m FIG. 2. The control frame 150 is the pn- front view of the clement displays the icon label and the 

mary context for mteracting with the user mterface of j ^^^^^^^ be detennined from the icon on its 

Ac present invenhon. and is. primarily, comprised of ^^ j^q 5) ^ 

four comrol panels which surround a vanety of subwui- ^ iUustraf es the relative length in time the ele- 

dows. mee ofthe panels are yu.ble to the user M all 20 ^^^^ ^ ^j^^ f,^^ jU^^^^,^ 

tunes, and the bottom panel (not shown) is displayed on ^^^^^^ ^^^^^^ relative lengths. The 

an as needed basis The mode specific action pane 1S2 element/resource may comprise several indMdual ele- 

compnses, m the displayed embodiment, a plurality of * w v ji j • * j i 

direct action buttons^ or icons, which change with the "^^'^^ "'"^ compound de- 

program mode. As shown in the illustration of FIG. 6. 25 current users may group graphical ele- 

the mode specific action panel 152 comprises a visual "^^^^ usmg a graphics editor, . ,„ 

element 155, an aural element 158, a text element 162. a . ^? t<^P.^<^w 215 

graphic element 164. and a process element 166. Al- present mv«ition to position elements 

though buttons 155. 158. 162, IH and 166 are illus- «^^]>ve to tome by placmg them on a time hne. As will 

trated in FIG. 6, it will be appreciated that the mode 30 ^ more detail below, placmg an clement 

specific action panel buttons/icons change to support ^" » referred to as the event horizon" 220 mte- 

and reflect the current activities of the user for a partic- K^**" ^« element into the overall time line for the 

ular venue. production and results in the display of time data in the 

The control frame 150 further includes a major mode ^« identified in TIG. 6 as 218. Moving an clement 
panel 170, which comprises an edit button 172, a graph- 33 construction region 200 to the event horizon 220 
ics button 174, an audio button 176, a plan button 180, results m the element being assigned a time assignment 
and a set up button 182. It will be noted that although ^ ^« ^icw port 215. It will be noted that the view port 
throughout this description icons, buttons, and the like 215 corresponds to the top view port view of an elc- 
are described, that the reference to buttons, icons, etc. n^^nt in the thrccKiimensional representation of re- 
represents any class of displayed items which result in 40 sources described with reference to FIG. 4. 
some executoble action when chosen by a user. There- Referring now to FIG. 7, assume for sake of example 
fore, although an edit button 172 is disclosed as part of * user desires to define an element using the present 
the control frame 150, it will be appreciated to one invention's user interface. An element attribute box 250 
skilled in the art that the edit button 172 may comprise ^ displayed by computer 20 depicctcd in FIG. 2 once a 
an icon in the form of some edit feature or the . like 45 selection identified as "element" 252 is chosen by a user, 
which achieves the same result In the presently pre- Although in the presently preferred embodiment, de- 
ferred embodiment, the buttons comprising the major ^^^^ 252 comprises a pull down menu (not shown) 
mode panel 170 are always present for the user no mat- having a plurality of items for selec^on, conceptually 
tcr which venue or other option is selected. the clement 252 is selected by a user through the use of 

In general, the major mode panel 170 permits a user SO ^ cursor control device (see for example, U.S. Pat. No. 

to access different venues than the venue currently RE32633). Although a cursor control device, such as a 

displayed. The specific b\ittons/icons used in the m^jor "mouse" may be used to select element 252, it will be 

mode panel 170 is a function of the particular project in appreciated Uiat the actual selection may be made using 

which the user interface is implemented. A menu bar & variety of display input devices known in the art It 

panel 190 generally displays labels for pull down menus. S3 will also be appreciated by one skilled in the art, that 

Standard labels such as "organize", "modify", etc. are other mechanisms for selecting element 252 and the like 

provided and are present no matter which venue or u;e known. Accordingly, the particular mechanism for 

resource is accessed. Other context specific menu labels selecting functions, items, icons and the like within 

wilt be displayed such as **eleroent", "edit style", "utili* control frame 150 will not be described in detail in this 

ties", "setup", etc. are provided for specific applica- 60 Specification. 

tions. Once element 252 is selected, computer 20 illustrated 
The application specific construction area 200 com- in FIG. 2 displays the element attribute box 250 as 
prises an application area of the display for a selected shown in FIG. 7. A user then either selects from a pre- 
program mode, and is available for subwindows, user printed list of elements, or defines elements within the 
views in the display of other elements of work such as 63 element attribute box 250, the nature of the resource 
time lines, scripts, scores, preview monitors, etc. As required for the particular production. Examples of 
shown in FIG. 6, construction area 200 is designated as such element attributes which a user may select include, 
a resource manager. As will be described, a top view but are not limited to the following: 
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VISUAL ELEMENT ATTRIBUTES TEXT ELEMENT ATTRIBUTES 

Label Label 

User supplied Element identification, (e.g.: Video j User supplied Element identification. 
Tape Recorder, etc.) j^^^^ 

Source Device Title of Text type and Document. (SCRIPT. OPEN- 

Assignmeni of device: (e.g.: PI, P2, RECORD, etc.) TITLE, CHARACTER GENERATOR, ETC) 

Identification '° Revision 

Alpha-Numeric material Identification: e.g.: reel *=!"T«"' T"* Previous 

number, reel label, etc. lab roll number, etc. "nfonnat'on- 

Scene/Take Information ,5 FUe Types 

Content Scene and Take identification. "^^^ "^^'^ 

Scene Take file name 



useable fonn. 

Associated Data 



Codes 

^. _ , ■ _ . . ^ _ . 20 Any previous data files associated with creating the 

Time Code and type. Ongm Tune Code and type. current file. 
User bit Time Code. User bits. Content, Frame Num- 
bers. Edge Numbers. Code Numbers. Processing Path 

j^Q^gg Information on any previous Processing that applied 

25 ^ ^^^^ material. 

Associated Text Notes For Reel &/or Content. 

GRAPHIC ELEMENT ATTRIBUTES 

Label 

User assigned Priority levels for different versions. ^ser supplied Element identification. 

Processing Path Tit\c 

Information on any previous Processing that applies ^ user supplied description of the Graphic element 
to this material, (e.g.: DUB LEVEL, COLOR COR- 
RECTION, ETC.) Revision 

AUDIO ELEMENT ATTRIBUTES " . Graphic revision level and relevant pre- 



Label 

User supplied Element identification. 



vious revision information. 

File Types 



The Names and Types of files as the material exists in 
Source Device ^ useable form now. 

Type and Assignment of device, (e.g. ATR, DISC, Associated Data 

ETC.) PI, P2. RECORD, ETC. previous data files associated with creating the 

Identification current file. 

Alpha-Numeric material identification, (e.g. Reel Processing Path 

number. Reel label etc. Sound Roll Number, Label.) Information on any previous Processing that applies 

Scene/Take # ^ material. 
Content Scene and Take identification. 50 PROCESSING ELEMENT ATTRIBUTES 

Codes ^^^^ 

Time Code and type. Origin Time Code and type. '""^^^^^ identification. 

User bit Time Code contents. Frame Numbers. Device Identification 

Xracks Effects Device identification. 

Number and Numbers of Source Track Assignment Path 

KT.^ Video and/or Audio routing assignments, (e.g. 

60 CROSSPOINTS. KEY CHANNELS. ETC.) 



Associated Text Notes. For Reel &/or Content 
Priority 

User assigned Priority levels for different versions. 

Processing Path " ^.^^ ^.^^ ^^p^ ^^^^^ durations. Effects 

Information on any previous Processing that applies Source codes. Effects Edge Numbers for optical printer 
to this material, (e.g. Dub level, Equalization, etc.) outputs. 



Up Load/Down Load 
File input/output for created effects save and recall. 
Codes 



04/14/2004, EAST Version: 1.4.1 



Processing 

Effects types, (e.g. CUTS, FADES, DISSOLVES, 
WIPES, KEYS, DME. ETC.) 

Once the element attributes have been defined,, com- 5 
putcr 20 illustrate in FIG. 2 utilizes appropriate network 
connections over network 44 to the various resources, 
such as the VTR 52, music synthesizer 55, Audio tape 
recorder 60, Production switcher 62, etc. to access the 
resource via the user interface. Accordingly, a direct 10 
connection via computer 20 has been created between 
the user interface comprising the control frame 150 as 
displayed on the display 50, and the particular element- 
/resource coupled through the network interface 42, 
Refering to FIG. 7, within the top view port 215, time 15 
elements corresponding to the particular resource have 
additional information that is revealed in the time tine 
associated with the top view port 215. Tracking buses 
265 provide additional information regarding the re- 
cording of audio channels 1 through 4, and a video 20 
channel 1. In practice, a source tape machine (not 
shown) supplies audio to a tape record machine wherein 
the channels are coupled to one another. It has been 
found that it is quite useful to display audio channels in 
the time view port 215, in order to correlate the audio 25 
channel and time interval versus resource. 

Once the element attributes have been defined, an 
element representing a resource is created based upon 
those attributes, and displayed within construction area 
200 of the control frame 150. Referring now' to FIG. 8, 30 
the present invention's user interface is illustrated 
wherein a plurality of elements identified as Record 
VTRI 300, Scene I 310, scene II 312, Dissolve 314, 
Open Title 316, and Key 320 are showa As illustrated 
in FIG. 8, an element such as Record VTRI 300 in- 35 
dudes an icon image (for example, the planet Earth in 
FIG. 8) which describes some aspect of the element for 
identification purposes. Viewing elements disposed in 
construction area 200 normally corresponds to viewing 
a venue and associated resources through a front view 40 
port as shown previously with respect to FIG. 4. An 
clement, for example. Record VTR 300, may be moved 
within the construction region 200 at will by a user 
through the use of an appropriate command sequence, 
or by simply dragging the element around the construe- 45 
tion area using a cursor control device such as a mouse. 
However, using the teachings of the present invention, 
once an element such as Scene I 310 is brought, 
dragged, or otherwise manipulated downward to the 
event horizon 220, the element is automatically given 50 
time significance which is represented along the time 
lines of the top view port 215. 

As illustrated in FIG. 8, the event horizon 220 com- 
prises a horizontal bar with arrows 221 and 222 at each . 
of its opposite ends. By placing a cursor (not shown) 55 
over arrows 221 or 220, and presenting computer 20 
with an activation signal, resource elements such as 
scene I 310, scene II 312, Dissolve 314, etc. may be 
moved left or right, respectively, and other elements . 
may be viewed which are currently not visible on the 60 
event horizon in the control frame 150. The use of ar- 
rows 221 and 222 permits a user to scan through ele- 
ments disposed on the event horizon 220 and view the 
elements not only in relative position, but in relative 
time. This view corresponds to that of a user 100 in 65 
FIG. 3 scamiing the resources in that Figure, such as 
special effects 90. editing 92, video 95, etc. In addition, 
it will be appreciated that the relative position of the 
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element may be changed by simply "dragging" an ele- 
ment such as scene I 310 off of the event horizon 220, 
moving other elements into that time slot along the 
event horizon, and replacing scene I 310 at some other 
location along the event horizon. A redistribution of the 
element's relative position along the event horizon 
would correspond in FIG. 4 to, for example, swapping 
element 110 for element 116 and vice versa. 

Once an clement is placed upon the event horizon 
220, position data relative in time to other elements is 
illustrated along the time lines of the top view port 215 
as shown. Conceptually, the reader is directed to FIG. 
4 which illustrates in three dimensions the placement of 
elements relative to one another in time. However, due 
to the limitations of display 50, depicted in FIG. 2 the 
time view port 215 is utilized to display time along the 
± Y direction, with time To being at the lower portion 
of the display as illustrated in FIG. 8. In addition, as 
shown in FIG. 8, the number of versions of, for exam- 
ple, scene I 310, is also displayed as versions 351 
through 354. It will be appreciated by the reader that 
the display of an element such as scene I 310 corre- 
sponds to the prior description of a resource having 
multiple versions which may be activated by selecting 
(for example, by placing a cursor over version 354 and 
providing an activation signal) a version such that the 
version is '*run" within the window of scene I 310. 
Accordingly, user may view the entire version of scene 
I which has been selected, within the icon window 
comprising the scene. In general, in the present and 
preferred embodiment, and throughout this specifica- 
tion, placing a cursor over an icon or other executable 
function and "double clicking" using a cursor control 
device such that two consecutive signals are provided 
to the computer 20 depicted in FIG. 2 executes the 
function which has been selected, and, more particu- 
larly^ reveals any attributes and/or contents of the icon. 
Double clicking on a time function such as time block 
400 for scene I (FIG. 8) may be configured such that 
time code (i.e. SMPTE) is displayed. More particularly, 
in the present example, SMPTE time code for the be- 
ginning and end of each version within scene 1 310 may 
be displayed within the time line view port 215. 

Referring now to FIG. 9, control frame 150 is illus- 
trated in which a new scene I 400 has been "dragged" 
upward into the construction area 200. As illustrated in 
FIG. 9, once an element, in the present example scene I 
400, is moved off of the event horizon 220, timing infor- 
mation viewed through top view port 215 correspond- 
ing to scene 1400 is no longer displayed. Elements such 
as scene II 404, dissolve 406, open title 408 or key 410 
may be repositioned along the event horizon 220 and/or 
modified in terms of time sequence as viewed through 
top view port 215, relative to one another. Alterna- 
tively, and as shown in FIG. 9, clement attribute box 
250 may be selected and the attributes of scene I 400 
may be modified, or an entirely hew clement may be 
defmed, to replace scene I 400 along the event horizon 
220. 

It will be appreciated that a user utilizing the inter- 
face comprising the present invention defines elements 
in the construction area 200 by specifying attributes of 
the element in the element box 250. In addition, multiple 
elements may be created within the construction area 
200. The created elements may then be selectively 
dragged to the event horizon 220 in an arrangement and 
order selected by the user. It will further be appreciated 
from the above discussion by one skilled in the art that 
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the present invention*s user interface permits the utiliza- 
tion of resources within the system iUustratcd in FIG. 2, 
permits selective modification of the resources, and 
through the use of the interface of the present invention 
provides a consistent interface for the production of an 5 
audio visual work. The user interface of the present 
invention, through the use of a common control frame 
150 as displayed on display 50, allows artists, musicians, 
and other media professionals to create, modify and 
rearrange resources comprising a production with flexi- 10 
bility heretofore unknown in the art. The present inven- 
tion's concept of venues, and the ability to operate on 
resources in three dimensions, provides a user with 
flexibility not present in any prior user interface for 
computer display systems, as well as multi-media pro- 
duction systems known in the past. 

The present invention has been described and illus- 
trated with reference to the Figures as applied to a 
display 50, and using input devices, such as digital pad 
36, trackball 40 and keyboard 38 as shown in FIO. 2. 
However, the teachings of the present invention may be 
applied to numerous other display devices and input 
mechanisms. For example, the present invention may be 
practiced using what is known as 'Virtual reality** input 
devices, such as but not limited to, a data input glove, 
body glove input device, etc. In addition, the present 
invention may be utilized with eye goggle displays 
which are worn by a user and coupled to the computer 
display system via fiber optics, wires and the like. When 
the present invention is utilized in conjunction with a 
virtual reality system, the user interface of the present 
invention would be viewed by a user through input 
goggles as being suspended in space. Interaction with 
the interface by the user may be done using an input 35 
glove or other virtual reality device worn by the user. 
Accordingly, it will be appreciated that the user inter- 
face of the present invention is not limited to conven- 
tional input or display devices. The reader is referred to 
the following references for a further description of 40 
existing and proposed virtual reality systems. Computer* 
tied Reality Comes of Age, NASA Tech Briefs, page 10, 
August 1990 (Vol. 14, number 8); Iwata, Artificial Real- 
ity with Force—Feedback; Development of Desktop Vir- 
tual Space with Compact Master Manipulator^ ACM 45 
SIGGRAPH, August 1990 CVol. 24, number 4); Nash, 
Our Man in Cyberspace Checks out Virtual Reality, Com- 
puterworld, Oct. 15, 1990; Daviss, Grand Illusions, 
Discover, June 1990. 

While the present invention has been described in 50 
conjunction with a few specific embodiments identified 
in FIGS. 1 through 9, it will be apparent to those skilled 
in the art that many alternatives, modifications and 
variations in light of the foregoing description arc possi- 
ble. Accordingly, the invention is intended to embrace 5s 
all such alternatives, modifications and variations as 
may fait within the spirit and scope of the invention as 
disclosed. 

We claim: 

1. A computer controlled display system including at 60 
least one Central Processing Unit (CPU), said CPU 
coupled to a display for displaying data, and user input 
means, said display system further coupled to a plurality 
of system resources having defined attributes, said dis- 
play system comprising: 65 
a user interface for display by said display means, said 
user interface for displaying representations of 
resources coupled to said display system with 
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which a user interacts through said user input 
means; 

said representations of said resources comprising , at 
least three dimensions and being arranged in a 
venue of at least three dimensions, each of said 
dimensions corresponding to an attribute of said 
resources, said venue may be viewed using said 
interface from a plurality of view ports, such that 
viewing said representations of said resources from 
different view ports results in the display of differ- 
ent attributes of said resource^ 

said representations of said resources being arranged 
in said venue such that each of said representations 
is disposed relative to one another in time and 
space within said venue; 

manipulation means coupled to said user input means 
for selectively positioning said representations of 
said. resources within said venue. 

2. The display system as defmed by claim 1 wherein 
said venue is three dimensions and each of said repre- 
sentations of said resources comprise three dimensions. 

3. The display system as defmed by claim 2 wherein 
said venue may be viewed from six view ports, namely, 
a top, left, right, rear, bottom and front view port. 

4. The display system as defmed by claim 3 wherein 
viewing said venue from a top view port reveals the 
relative positions of each of said three dimensional rep- 
resentations of said resources relative in time to one 
another. 

5. The display system as defmed by claim 4 wherein 
viewing said venue from a front view port reveals an 
icon identifying the type of resource each of said repre- 
sentations represent. 

6. The display system as defined by claim 5 wherein 
viewing said venue from a side view port reveals ver- 
sions of said resource and the lengths of said versions 
relative to one another. 

7. The display system as defined by claim 6 wherein 
viewing said venue from a front view port further re- 
veals version activation buttons, one of said buttons for 
each of said versions, such that said user, using said user 
input means may selectively activate said versions. 

8. The display system as defined by claim 7 wherein 
in the event said user activates one of said versions, said 
version is executed within said representation of said 
resource, and may viewed through said front view port 
of said venue on said display. 

9. The display system as defined by claim 6 wherein 
said user interface includes interface generation means 
coupled to said CPU for generating and displaying a 
control frame using said display means to display se- 
lected view ports of a venue, said control frame includ- 
ing a plurality of command options which may be se- 
lected by said user using said user input means. 

10. The display system as defined by claim 9 wherein 
said control frame fiirther includes a first area for defin- 
ing said attributes of said resources and displaying said 
representations of said resources once said attributes are 
defined. 

11. The display system as defined by claim 10 
wherein said first area displays said representations of 
said resources in a venue initially from a first view port. 

12. The display system as defined by claim 11 
wherein said user may selectively change view ports 
from said first view port by selecting one of said com- 
mand options. 

13. The display system as defined by claim 10 
wherein said control frame further includes a second 
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area for displaying said top view port of said venue, 23. The display system as defined by claim 22 

such that timing data representing relative time associ- wherein said command options of said control frame 

ated with said resources is displayed in said second area. further includes a plurality of mode specific action op- 

14. The display system as defined by claim 13 tions on the periphery of said control frame, 
wherein said control frame further includes a event 5 24. The display system as defined by claim 23 
horizon bar, such that the placement of a representation wherein said representations of said resources are ar- 
of a resource on said bar results in said Hming data being ranged in said venue such that each of said resources is 
displayed in said second area of said control frame. disposed relativie to one another in time and space 

15. The display system as defined by claim 14 within said venue, 

wherein selecting an Element command option results 10 25. The display system as defined by claim 24 further 

in the disptoy of an element attribute box for defining including manipulation means coupled to said user input 

said resource attributes. means for selectively positioning said representations of 

16. The display system as defined by claim 15 said resources within said venue. 

wherein said representations of said resources may be 26. The display system as defined by claim 24 

selectively placed on said event horizon bar, thereby wherein said venue is three dimensions and each of said 

altering the relative placement of said representations in representations of said resources comprise three dimen- 

said venue in time and space. sions. 

17. The display system as defined by claim 16 27. The display system as defined by claim 26 
wherem said command options of said control frame wherein said venue may be viewed from six viewports, 
further mcludes a plurality of mode specific action op. ^gly. a top, left, right, rear, bottom and front view 
tions on the pnenphery of said control frame. poj^ 

18. The display system as defined by claim 17 28* The display system as defined by claim 27 
wherem said command options of said control frame wherein viewing said venue from a top view port re- 
fiirther mcludes a plurahty of major mode options on ^cals the relative positions of each of said three dimen- 
the periphery of said control frame sional representations of said resources relative in time 

19. The display system as defined by claim 18 another 

wherein said first view port initially displayed com- 29, The display system as defined by claim 28 

pnsjes the front view port . , wherein viewing said venue from a front view port 

20. A computer controlled display system mcludmg .^veals an icon identifying the type of resource eachof 
at least one Central Processing Umt (CPU), said CPU representations reprint. . 

coupW to display means for displaymg data on a 30; ^he display system as defined by claim 29 

f^l^^^^^'r^"^" input means, said display system ^^erein viewing said venue from a side vL port re- 

further coupled to a plurality of system resources hav- ^^^-^^^ ^^^^^ 

mg defined attributes, said display system comprising: 33 ^^^3^^^ ^^^^^^ ^ 

a user mterface on said screen of said display for 3, ^ j ^ ^ 

irsrs^s ^srr^sj^ xs',jc:£H"t f.-j^ n^--^ 

through said user input means; f"^^^ ^ ^"«^"t' ^d 

said user interface including interface generation 40 ^d^^'i^T ."^^ 

means coupled to said CPU for generating and «"d user mput means may selectively activate said 

displaying a control frame on said display, said , . ^ . . 

control frame including a plurality of command .^.^ ^'P^^^ '^y^!"* *^ 31 

options which may be selected by said user using wherem m the event said user acUvates one of said 

said user input means, and further including a event 45 Y^^'^ns, said version is executed within said represcnU- 

horizon bar, such that the placement of a rcprcsen- resource, and may view^ through said 

tation of a resource on said bar using said user input ^'^^ "^^^^ display, 

means results in predetermined data being dis- J^^ display system as defined by claim 32 

played in said control frame and said rcprescnU- wherein said first area displays said representations of 

tions of said resources being disposed relative to 50 resources in a venue initially from a first view port, 

one another in time; ^* display system as defined by claim 33 

said representations of said resources comprising at wherein said user may selectively change view ports 

least three dimensions and being arranged in a said first view port by selecting one of said com- 

venue of at least three dimensions which may be options. 

viewed using said user interface from a plurality of 55 The display system as defined by claim 34 

view ports, such that viewing said representations wherein said control frame ftmher includes a second 

of said resources from different view ports results ^ca for displaying said top view port of said venue, 

in the display on said screen of different attributes *^ch that timing data representing relative time associ- 

of said resources. ated with said resources is displayed in said second area. 

21. The display system as defined by claim 20 60 ^* display system as defined by claim 35 said 
wherein said control frame further includes a first area representations of said resources may be selectively 
for defining said attributes of said resources and display- placed on said event horizon, thereby altering the rcla- 
ing representations of said resources once said attributes tive placement of said representations in said venue in 
arc defined. time and space. 

22. The display system as defined by claim 21 65 37. The display system as defined by claim 36 
wherein selecting an Element command option results wherein said command options of said control frame 
in the display of an element attribute box for defining further includes a plurality of major mode options on 
said resource attributes. the periphery of said control frame. 
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38. The display system as defined by claim 37 44. The method as defined by claim 43 wherein said 
wherein said first view port initially displayed com- venue is three dimensions and each of said reprcscnta- 
prises the front view port. ^^O"^ resources comprise three dimensions. 

39. In a computer controlled display system including method as defined by claim 44 further includ- 
at least one Central Processing Unit (CPU), said CPU ^ ?^^P of selectively viewing said venue from one 
coupled to a display for displaying data on a screen of ^^'^^ viewports, namely, a top, left, right, rear, bottom 

.. .. , I • * J J- 1 and front view port, 

^ud duiplay and user mput means, srnd d^phy system ^ j^^^ ^^^^ ^ 45 ^^^^^^.^ ^j^^. 

further coupled to a plurality of system resources hav- -^^ ^g„„^ f^^^ 3 ^^^^^ 
mg detmed atinbutes, a method for mampulatmg and ,0 positions of each of said three dimensional representa- 
displaying representations of said resources on said tions of said resources relative in time to one another, 
display, comprising the steps of: 47. The method as defined by claim 46 wherein view- 
providing a user interface on said screen of said dis- ing said venue from a front view port reveals an icon 

play for displaying representations of resources identifying the type of resource each of said representa- 

coupled to said display system with which a user 15 tions represent. 

interacts through said user input means; 48. The method as defined by claim 47 wherein view- 
generating and displaying a control frame on said said venue from a side view port reveals versions of 
display using interface generation means coupled resource and the lengths of said versions relative to 
to said CPU. said control frame including a plural- ^ ^"f^*??^^ ^ ^ , . . . 
ity of command options which may be selected by . TT^e method as defmed by claim 48 wh^^^ 

. said user using said user input means, and further ^^.^^ ^'fZ^^^^^^ 

. / , , version activation buttons, one of said buttons for each 

mcluding a event horizon bar, such that the place- ^^^^ ^^^j^^g^ ^^j^ ^j^at said user, using said user input 

ment of a representation of a resource on said bar means may selectively activate said versions, 

using said user input means results in predeter- 25 50. The method as defined by claim 49 further includ- 

mined data being displayed in said control frame, ing the step of said user activating one of said versions, 

and said represenUtions of said resources bcmg said version then being executed within said representa- 

disposed relative to one another in time; tion of said resource, viewed through said front view 

said representations of said resources comprising at port of said venue on said display, 

least three dimensions and being arranged in a ^1. The method as defined by claim 50 wherein said 

venue comprising at least three dimensions which displays said representations of said resources 

may be viewed using said user interface from a ^ f^"^^ ^^j^^y fr?^ * "^^"f" 

plurality of view ports, such that viewing said rep- . tncthod as defined by claim 51 further mclud- 

r -J r j fr * • mg the Step of said user Selectively changmg View ports 

resentations of said resources from different view r • j c * • ». 1 a- 

^ 1^ - .1. ^- t -J i- 35 from said first view port by sclcctmg one of said com- 
ports results m the display on said screen of differ- ©and options 

ent attributes of said resources. 53 i^e method as defined by claim 52 whcrcm said 

40. The method as defined by claim 39 wherein said control frame further includes a second area for display- 
control frame further includes a first area for defining ing said top view port of said venue, such that timing 
said attributes of said resources and displaying represen- 40 data representing relative time associated with said 
tations of said resources once said attributes are defined. resources is displayed in said second area. 

41. The method as defined by claim 40 further includ- 54. The method as defined by claim 53 fiuther includ- 
ing the step of selecting an Element command option uiS the step of selectively placing said representations of 
and displaying an element attribute box for defining said resources on said event horizon, thereby altering 
resource attributes. relative placement of said representations in said 

42: The method as defined by claim 41 wherein said venue in time and space , . ^ . 

command options of said control frame further includes ^ defined by clami 54 whercm said 

. r J *L command options of said control frame further mcludes 

a plurality of mode specific action options on the pe- ^ ,.i„«ii*„ Xf «^«v*«. 

rir^^^ «aJH ^r^^frZ fra«i^ *^ & pluTaUty of ffiajor mode options on the periphery of 

nphery of said control frame. ^ control frame. 

43. The method as defined by claim 42 wherein said 5^. xhc method as defined by claim 55 wherein said 

representations of said resources arc arranged in said first view port initially displayed comprises the front 

venue such that each of said resources is disposed rcla- view port, 

tive to one another in time and space within said venue. • « • • • 

55 
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